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ABSTRACT 

The  vast  majority  of  software  engineers  are 
conscientious  professionals  who  work  hard 
to  convince  themselves  that  their  software 
does  the  right  thing  and  works  correctly. 
But  they  almost  never  write  down  what  they 
do  so  that  they  can  convince  someone  else. 
The  work  they  do  is,  in  essence,  verification 
and  validation.  Part  of  the  problem  is  that 
developers  often  don’t  think  of  what  they  do 
as  “V&V”,  they  may  or  may  not  be  focused 
on  the  M&S  application  requirements,  and  if 
it’s  not  documented  then  they  can’t  get 
credit  for  it,  anyway. 

This  paper  describes  a  cost-effective  VV&A 
approach  centered  on  the  capability, 
accuracy  and  usability  of  M&S.  This 
approach  focuses  V&V  activities  on 
accreditation  requirements  by  formalizing  an 
intended  use  statement  by  the  ultimate  M&S 
user.  The  approach  also  facilitates  making 
best  use  of  activities  that  already  are 
ongoing  during  the  development  of  the 
software.  We  identify  some  useful  tips  for 
low  cost  informal  documentation  of  V&V 
information  required  to  support  an 
accreditation 
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INTRODUCTION 

Many  program  managers  in  the  defense 
acquisition  community  think  that 
verification,  validation  and  accreditation 
(VV&A)  of  models  and  simulations  (M&S) 
is  too  hard,  costs  too  much  and  takes  too 
long.  Most  of  the  time  they  are  thinking  of 
extensive  Independent  Verification  and 
Validation  (IV&V)  efforts;  the  IV&V 
practitioner  rightly  wants  to  make  sure  that 
the  M&S  he  or  she  supports  have  a  well 
documented,  disciplined  process  of  V&V 
applied  to  them. 

However,  in  our  experience  most  M&S 
developers  are  already  doing  a  lot  of  the 
right  things  in  V&V,  but:  (1)  those  same 
developers  don’t  see  much  of  what  they’re 
doing  as  “V&V”,  so  they  under-report  what 
they  do.  (2)  Most  developers  don’t  write 
down  what  they  did  in  any  retrievable 
fashion,  or  they  leave  out  key  pieces  of 
information.  This  is  exacerbated  by  the 
extensive  use  of  personal  computers,  which 
has  nearly  eliminated  the  use  of  engineering 
notebooks,  and  creative  people  often  detest 
documenting  anyway.  (3)  Pressure  to 
produce  a  product  on  time  and  within  budget 
often  means  that  if  program  managers  fund 
any  V&V  activities,  funding  for 
documenting  them  falls  off  the  table. 
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We  have  helped  a  number  of  acquisition 
programs  to  develop  a  cost-effective  VV&A 
approach,  using  the  results  of  work  that  they 
were  already  doing.  All  of  these  efforts  did 
require  some  additional  effort,  but  we  were 
in  every  case  able  to  leverage  previous  work 
extensively.  Many  of  these  customers  were 
doing  considerable  V&V,  but  it  was  not 
focused  on  their  application  and  it  wasn’t 
being  documented.  We  have  helped  them  to 
focus  their  efforts  on  the  requirements  of 
their  application,  and  to  document  their 
ongoing  V&V  efforts  in  ways  that 
minimized  any  additional  effort  while 
capturing  their  critical  V&V  results. 

ACCREDITATION  PROCESS 

Figure  1  illustrates  the  process  that  the  Joint 
Accreditation  Support  Activity  (JASA)  uses 
to  support  a  program’ s  accreditation  of 
models  and  simulations  (and  sometimes, 
physical  simulators). 


Accreditation 


Figure  1.  Accreditation  Process 

The  nature,  scope  and  depth  of  information 
necessary  to  accredit  a  simulation  depends 
partly  on  what  the  simulation  is  being  used 
for  and  partly  on  how  much  risk  is 
associated  with  its  use  for  that  purpose. 
Accreditation  is  a  statement  of  confidence  in 
the  credibility  of  the  simulation  for  a 
specific  application  (it  is  not  a  statement  of 
general  credibility  for  any  application 
whatsoever.) 


M&S  requirements  and  acceptance  criteria 
are  determined  by  a  combination  of 
application  analysis  and  risk  analysis. 
Application  analysis,  or  use  analysis 
establishes  the  specifics  of  how  the 
simulation  will  be  used,  and  which 
simulation  functions  and  outputs  are 
important.  Risk  analysis  establishes  the 
level  of  credibility  required,  and  which 
aspects  of  simulation  credibility  are  essential 
to  accreditation.  From  those  two  activities 
we  can  determine  V&V  requirements.  By 
comparing  ongoing  V&V  activities  with  the 
requirements  for  V&V,  we  can  develop  a 
tailored  VV&A  plan  specific  to  the 
application.  When  the  planned  VV&A 
activities  are  completed,  an  accreditation 
recommendation  can  be  made  to  the 
Accreditation  Authority,  who  makes  the 
final  accreditation  decision.  A  key  element 
of  the  process  is  to  determine  any  limitations 
on  the  use  of  the  simulation  based  on  the 
documented  results  that  are  available,  and 
any  proposed  workarounds  in  areas  where 
the  simulation  may  not  be  acceptable. 
Ultimately,  the  accreditation  provides  an 
assessment  of  the  risks  of  using  the 
simulation  for  the  application. 

The  key  to  making  this  process  cost- 
effective  is  to  ficus  the  required  activities 
where  they  most  make  sense.  In  the  first 
phases  of  the  process,  the  focus  needs  to  be 
on  the  intended  use  of  the  simulation;  in 
fact,  the  entire  purpose  of  the  application 
analysis  and  Intended  Use  Statement  is  to 
develop  M&S  requirements  and  acceptance 
criteria  for  the  specific  use;  these  will  be  the 
criteria  against  which  the  M&S  will  be 
judged.  In  the  last  phases,  the  focus  is  on 
accreditation:  that  is,  does  the  accreditation 
authority  have  enough  information  to  make 
a  decision  that  the  M&S  meets  those 
acceptance  criteria? 


And  that  is  where  the  rubber  meets  the  road. 
Too  often  programs  have  done  considerable 
work  to  ensure  that  their  simulation  has  the 
capability,  accuracy  and  usability  they 
require,  but  they  have  not  made  sure  that 
their  work  has  been  documented.  So  they 
did  the  work,  but  they  don’t  get  the  credit, 
thus  running  the  risk  of  not  being  accredited 
by  the  ultimate  user. 

What  JASA  has  done  for  a  number  of 
customers  is  to  review  their  VV&A  plans, 
results  and  especially  their  documentation  of 
VV&A  activities.  We  have  consistently 
seen  that  programs  are  conscientiously 
tracking  the  credibility  of  their  M&S  tools. 
However,  they  have  almost  uniformly  not 
captured  that  work  in  documents,  or  even 
informally  in  lab  notes,  viewgraph 
presentations,  etc. 

We  have  found  that  it  often  is  not  necessary 
to  expend  resources  formally  documenting 
VV&A  results.  Informal  documentation  is 
often  sufficient  for  the  purpose  of  providing 
evidence  of  M&S  credibility,  as  long  as  it  is 
readily  retrievable  and  traceable  back  to  the 
M&S  version  used  and  the  personnel 
involved.  The  following  are  some  tips  on 
how  to  make  those  informal  records 
meaningful  to  an  accreditation  assessment. 

INCREASING  THE  VALUE  OF 
INFORMAL  RECORDS 

Based  on  our  experience  with  a  number  of 
programs,  we  have  come  up  with  some 
simple  ways  to  increase  the  value  of  any 
existing  information  you  may  have  for 
supporting  an  accreditation  decision,  as  well 
as  some  thoughts  on  how  to  better  capitalize 
on  planned  activities  in  the  future.  These 
ideas  have  been  used  successfully  by  a 
number  of  DOD  acquisition  programs  over 
the  last  several  years. 


Ideas  for  Increasing  the  Value  of  the 
Existing  Record 

1.  As  a  guiding  principle,  focus  on  content 
rather  than  aesthetics.  With  a  limited 
amount  of  time,  it  is  much  more  important 
to  write  the  information  down  clearly  in  a 
place  where  it  can  be  retrieved  easily  than  it 
is  to  get  fancy.  Write  information  by  hand 
onto  meeting  agendas  or  onto  a  hardcopy 
of  the  viewgraphs  from  meetings. 

2.  Gather  technical  resume  sheets  for  each 
person  on  the  development  team  and  keep  it 
with  the  team  records.  Include  information 
on  the  technical  qualifications  of  each 
person  (training,  experience)  as  well  as  their 
major  areas  of  emphasis  or  responsibility  in 
developing  the  simulation.  We  suggest  you 
also  include  information  on  people  who  are 
referenced  in  your  files  or  working  notes. 
The  qualifications  of  the  team  members  help 
to  build  confidence  in  the  product. 

3.  Look  through  the  briefings  you  have  on 
hand,  find  incomplete  references  and  fill  in 
the  missing  information.  Write  the 
information  on  a  hardcopy  of  the  viewgraph 
and  keep  it  in  the  program  files.  For 
example,  if  there  is  a  reference  to  an  author 
(Skolnik,  for  example),  write  down  the  title 
of  the  book  or  paper  or  internal  memo  and 
the  date.  Noting  the  pages  from  that 
reference  that  contain  the  relevant 
formulations  would  be  a  good  practice. 
Particularly  for  internal  memos,  some  hint 
on  where  to  get  a  copy  of  the  reference 
would  be  helpful.  If  you  have  the  document, 
make  a  copy  of  the  cover  or  title  page  and 
the  relevant  sections  and  put  it  in  the  file 
with  the  viewgraphs. 

4.  For  data  plots  which  contained  more  than 
one  trace,  include  a  key  which  indicates 
which  line  is  what.  If  your  team  uses  a 
consistent  convention,  (old  version  is  solid 


line  and  new  version  is  dotted  line,  for 
example)  a  note  at  the  beginning  of  the 
record  that  describes  the  plotting 
conventions  would  be  useful  to  outside 
observers. 

5.  Also,  for  plots  in  your  existing  record,  go 
back  and  make  some  comments  directly  on 
the  plots  about  what  the  plot  is  showing, 
your  analysis  of  what  the  plot  means,  and 
any  inferences  you  can  make  about  the 
implications  for  credibility  or  usefulness  of 
the  simulation.  If  the  plots  show  good 
correlation  between  simulation  predictions 
and  field  data  or  expert  opinion,  or  the  plot 
indicates  a  problem  that  you  followed  up 
and  corrected,  this  is  important  evidence  of 
the  correctness  of  the  simulation  that  you 
should  get  credit  for.  But,  you  don’t  get 
credit  for  good  correlation  if  people  don’t 
know  what  they’re  looking  at. 

6.  Look  over  the  records  and  jot  on  the 
viewgraphs  which  version  of  the  simulation 
the  information  is  related  to  and  the 
approximate  date  the  memo  or  viewgraph 
package  or  whatever  was  generated. 

7.  Look  over  the  records  and  jot  on  the 
viewgraphs  which  version  of  the  system 
hardware  components  the  module  is  related 
to  (for  example,  is  a  model  of  the  Inertial 
Measurement  Unit  (IMU)  generic  or  is  it 
tied  to  a  particular  version  of  the  real 
IMU?).  You  should  describe  the  hardware 
component  in  enough  detail  to  uniquely 
identify  it:  serial  number,  delivery  date, 
label,  etc. 

8.  Also,  jot  down  enough  about  test  cases 
(initial  conditions,  test  file  name,  etc.)  that 
the  plot  is  meaningful  to  someone  else. 


Almost  Painless  Ways  to  Increase  the 
Value  of  the  Records  You  Generate  from 
Here  On  Out 

1.  Put  the  title,  date,  point  of  contact,  and 
contact  info  (telephone,  email  address)  on 
the  cover  of  any  viewgraph  presentations  or 
reports. 

2.  Document  the  version  of  the  code  to 
which  the  presentation  applied  and  the 
version  of  any  hardware  components  whose 
behavior  or  characteristics  are  reflected  in 
the  code. 

3.  Take  a  moment  during  informal  review 

sessions  to  jot  down  the  essence  of  the 
discussion  about  the  presentation.  Were 
there  concerns  about  any  of  the  charts 
shown?  Questions?  Is  there  anything  in  the 
plots  that  doesn’t  look  right  that  needs 

follow-up?  Was  there  consensus  that  the 
design  looked  good,  or  the  plots  matched 
what  you  expected,  or  there  was  good 

correlation  between  the  simulation 

predictions  and  the  reference  data  (test  data, 
intelligence  estimates,  whatever)?  If  it  looks 
good,  make  a  comment  to  that  effect  and 
initial  and  date  it.  We  suggest  that  the 
others  in  the  review  also  initial  the  hardcopy 
of  the  viewgraph.  If  something  looks 

strange,  mark  it  on  the  hardcopy  and  jot 
down  who  is  going  to  follow  up.  Keep  a 
key  in  your  records  that  has  a  list  of  team 
members’  clearly  legible  names  and  a 
sample  of  each  person’s  initials.  For  more 
information  on  conducting  effective 
reviews,  see  also  (Kilikauskas,  2002)*. 

4.  Require  an  assumptions  and  caveats  slide 
in  presentations  on  design  or  results.  It  just 
takes  a  moment  to  put  it  down  on  a  slide,  but 
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it  may  be  impossible  to  recreate  the 
information  later. 

5.  Keep  a  logbook  of  team  meetings.  Jot 
down  notes  of  who  was  there,  topics  of 
discussion,  activities  (peer  review  of  design 
fixes  for  the  IMU,  etc.)  major  findings,  and 
action  items.  This  helps  establish  a  pattern 
of  informal  review  over  time. 

6.  Software  engineers  generally  show  day- 
to-day  results  to  each  other  informally  for  a 
sanity  check  outside  of  a  review  meeting. 
This  is  a  form  of  peer  review.  When  one 
person  looks  over  someone  else’s  design  or 
code  or  output,  have  them  jot  their 
comments  on  the  paper  and  date  and  initial 
it.  This  may  be  particularly  helpful  for  work 
done  by  engineers  with  less  experience. 
Outside  observers  generally  feel  better  about 
work  done  by  less  experienced  team 
members  if  they  see  evidence  that 
colleagues  with  more  experience  have  been 
giving  them  a  hand. 

7.  In  briefings  about  module  design  or 
results,  include  a  slide  which  summarizes 
who  has  who  looked  at  the  design  or  output 
plots  so  far,  their  comments/conclusions, 
and  why  their  opinion  means  anything.  This 
helps  to  build  evidence  of  checks  during  the 
development  process  and  preserves  any 
comments  or  conclusions. 

8.  Require  that  software  engineers  record 
the  source  of  the  algorithms  and  data  they 
use  in  developing  individual  software 
modules.  If  the  engineer  modified  the  data 
after  it  was  received  from  the  original 
source,  record  how  the  data  were  changed 
and  why.  This  should  be  standard 
information  in  a  viewgraph  presentation  on 
the  design  or  modification  of  a  software 
module.  Putting  this  information  in  the  code 
as  comments  ensures  that  the  data  source 


and  modification  and  transformation 
information  stays  with  the  code. 

9.  In  any  briefings  on  changes  to  a  baseline 
simulation,  a  slide  or  two  on  the 
implications  for  simulation  use  would  be  a 
very  nice  addition  to  the  description  of  the 
changes.  What  do  you  expect  to  be  the 
effect  of  the  change  on  usefulness  or 
credibility  of  the  simulation?  If  one  were  to 
compare  results  of  the  previous  version  with 
the  new  version  for  the  same  input 
conditions,  what  differences  would  you 
expect  to  see  in  the  major  output  parameters 
of  interest? 

10.  In  briefings  about  software  design  or 
results,  require  a  slide  or  two  which 
describes  efforts  made  by  the  engineer  to 
determine  that  the  conceptual  design  of  the 
module  meets  the  requirements,  that  the 
detailed  design  is  consistent  with  the 
conceptual  design,  that  the  code  actually 
implements  the  design,  and  that  there  are  no 
coding  errors. 

11.  Keep  a  file  of  things  presented  at  team 
meetings.  Include  agenda,  attendance  list 
(maybe  put  all  team  members’  names  on  the 
agenda  as  a  standard  practice  and,  on  the 
copy  of  the  agenda  you  throw  in  the  file, 
pencil  out  the  name  of  those  who  don’t 
attend  that  particular  meeting),  viewgraph 
packages  with  notes  about  any  reaction, 
discussion  items,  and  decisions  related  to  the 
viewgraphs. 

12.  Keep  a  library  of  the  source  materials 
that  are  referenced  in  documents  about  the 
simulation  (design  document  and  briefings, 
analyst’s  or  user’s  manuals,  etc.).  At  the 
very  least  keep  a  list  of  where  to  get  them  so 
that  you  can  retrieve  a  copy  if  you  need  to. 


13.  It  helps  the  team  to  be  more  consistent 
in  following  these  practices  if  you  do  a 
couple  of  things: 

a)  Develop  a  template  for  briefings 
that  includes  place  holders  for  the  types  of 
slides  mentioned  (title  with  name,  date  and 
contract  information;  assumptions  slide; 
implications  for  model  use;  source  of 
algorithms  and  embedded  data;  etc.) 

b)  Choose  a  meticulous  person  and 
put  him  or  her  in  charge  of  taking  notes 
during  the  meetings  and  keeping  a  file.  Be 
kind  to  that  person  and  reward  him  or  her 
for  doing  this  job  for  you. 

c)  The  team  lead  needs  to  enforce 
following  these  practices.  Reward  people 
for  doing  this,  and  punish  them  if  they  don’t 
(make  them  do  it  over  till  they  include  these 
things). 

SUMMARY 

JASA  has  supported  a  number  of  acquisition 
programs  with  cost-effective  accreditation 
planning,  application  analysis  to  develop 
intended  use  statements  and  M&S 
acceptance  criteria,  V&V,  and 
documentation.  We  have  demonstrated  that 
you  almost  never  have  to  start  from  scratch: 
but  you  do  have  to  document  the  results  of 
V&V  efforts  that  you’re  already  doing  so 
that  an  accreditation  decision  can  be  well 
substantiated.  Informal  documentation  can 
be  just  as  effective  as  formal  reporting 
standards,  as  long  as  the  documents  are 
retrievable,  accessible,  and  meet  the  needs 
of  the  accreditation  authority.  When  it 
comes  time  to  accredit  M&S,  \diy  not  get 
credit  for  what  you’re  already  doing, 
especially  if  you  can  do  so  for  almost  no 
cost  and  very  little  effort? 
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